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The  report  is  divided  into  two  parts.  The  first  concerns  the  work  on  (i)  numerical  methods  for 
first  order  nonlinear  equations  PDE,  deterministic  control  and  large  deviations,  (ii)  development 
of  the  Skorokhod  Problem  and  related  methods  for  analyzing  and  controlling  queueing  and 
service  networks,  and  (iii)  variational  and  stochastic  methods  in  image  recognition,  and  the  first 
PI  was  the  main  contributor.  The  second  part,  in  which  the  second  PI  was  the  main  contributor 
concerned  (i)  nonlinear  filtering,  (ii)  communications  systems,  and  (iii)  stochastic  algorithms  for 
adaptive  communicatids  systems. 

Part  1.  There  are  a  number  of  deterministic  optimal  control  problems  for  which  a  global 
approximation  to  the  value  function  is  needed.  For  example,  in  small  noise  risk-sensitive  and 
robust  nonlinear  filtering,  the  optimal  (robust)  filter  is  defined  in  terms  of  the  value  function 
for  a  calculus  of  variations  problem  in  which  the  variational  integrand  is  a  functional  of  the 
observation  process.  Other  examples  occur  in  computer  vision  and  related  areas,  where  the 
optimization  problem  appears  as  a  representation  for  an  unknown  shape  or  other  geometric 
object,  and  in  large  deviations,  where  the  solution  is  used  to  construct  an  approximation  to  an 
invariant  distribution.  In  these  examples  the  representation  usually  takes  the  form  of  a  calculus 
of  variations  problem,  although  in  some  cases  a  differential  game  representation  is  required. 
In  other  problems  a  global  approximation  is  needed  so  that  an  approximation  to  the  optimal 
control  can  be  defined  at  all  points  in  the  state  space.  Examples  from  this  category  are  recent 
formulations  of  robust  nonlinear  control,  in  which  the  value  function  is  defined  in  terms  of  a 
differential  game  rather  than  an  optimal  control  problem  (although  in  some  cases  it  can  also 
have  a  representation  in  terms  of  an  optimal  control  problem).  An  common  special  case  for  all 
these  problems  is  that  in  which  the  dynamics  are  linear  in  the  controband  the  cost  is  quadratic 
in  the  control  (although  general  nolinearities  as  a  function  of  the  state  are  allowed).  The  paper 
[1]  constructs  Markov  chain  approximations  for  this  important  class  of  deterministic  control 
problems  and  differential  games.  The  emphasis  is  on  the  construction  of  schemes  that  are 
easily  programmed  and  which  possess  a  number  of  highly  desirable  qualitative  properties.  In 
particular,  (i)  the  schemes  have  minimal  numerical  dissipativity;  (ii)  all  minimizations  required 
by  the  iterative  solvers  which  calculate  the  approximation  can  be  done  analytically;  and  (iii) 
the  iterative  solvers  converge  rapidly.  Computational  examples  are  given,  as  is  a  representative 
proof  of  convergence. 

The  development  of  numerical  methods  for  this  class  of  problems  is  very  active.  The  main 
competitor  with  our  algorithms  is  the  so-called  fast  marching  method.  This  method  has  been 
developed  extensively  by  Sethian,  but  in  fact  appears  in  an  earlier  paper  of  Tsitsiklis.  Our 
method  is  easier  to  code  and  applies  to  a  much  broader  class  of  problems.  In  terms  of  com¬ 
putational  performance,  the  computation  associated  with  the  fast  marching  method  scales  as 
ATlogA/',  where  N^Dn^  is  the  total  number  of  grid  points,  d  is  the  dimension,  and  1/n  is  pro¬ 
portional  to  the  discretization  in  each  coordinate  direction.  In  contrast,  the  algorithms  we  have 
constructed  typically  scale  as  N, 

Further  development  of  our  method  to  higher  order  accurate  algorithms  appears  in  [2]  and 
[3].  The  paper  [2]  proves  foundational  results  on  the  convergence  of  controls  and  gradients  of 
value  functions,  which  are  then  used  in  [3]  to  construct  and  prove  the  convergence  and  higher 
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order  accuracy  of  a  second  order  scheme.  The  computational  burden  of  this  higher  order  scheme 
also  scales  as  n. 

Other  work  carried  out  during  the  period  has  concentrated  on  the  theory  and  applications 
of  the  Skorokhod  Problem.  The  solution  to  the  Skorokhod  Problem  provides  the  correct  in¬ 
put/output  map  for  many  models  of  communication  and  queueing  systems,  and  could  be  con¬ 
sidered  the  analogue  of  the  linear  system  model  for  standard  systems  theory  (i.e.,  systems 
without  constraints).  For  many  models  are  currently  (formally)  used  for  which  little  is  known 
regarding  existence,  regularity,  and  approximation  properties  of  the  corresponding  input/output 
map.  The  paper  [4]  develops  a  comprehensive  approach  to  this  problem  that  is  based  on  ideas 
from  convex  duality.  The  limitations  of  standard  methods  are  identified  in  very  explicit  terms, 
and  new  techniques  for  analyzing  systems  not  covered  by  these  results.  Two  of  the  main  ap¬ 
plications  of  this  theory  have  been  to  the  development  of  control  techniques  for  queueing  and 
service  networks  and  to  analyzing  large  deviations  in  stochastic  networks  [5]. 

In  the  past  few  years  a  new  approach  has  emerged  for  analyzing  the  stability  properties  of 
constrained  stochastic  processes,  such  as  reflecting  Brownian  motion,  reflecting  diffusions,  and 
queueing  networks.  In  this  approach,  one  associates  with  the  stochastic  model  a  deterministic 
model  (or  a  family  of  deterministic  models),  and,  under  appropriate  conditions,  stability  of  the 
stochastic  model  follows  if  all  solutions  of  the  deterministic  model  are  attracted  to  the  origin. 
In  the  paper  [6]  it  is  shown  that  a  sharp  characterization  for  the  stability  of  the  deterministic 
model  is  possible  when  it  can  be  represented  in  terms  of  a  “regular”  Skorokhod  Map.  This 
characterization  is  as  explicit  as  the  classical  eigenvalue  characterization  of  stability  for  linear 
systems,  and  is  currently  being  used  in  the  explicit  construction  of  stabilizing  and  optimally 
stabilizing  service  and  routing  controls  for  networks. 

This  paper  [7]  contains  two  main  results.  The  first  is  a  variational  representation  for  the 
expectation  of  a  measurable  function  of  a  Hilbert  space  valued  Brownian  motion,  when  the 
function  is  uniformly  positive  and  bounded  from  above  and  the  Brownian  motion  has  a  trace 
class  covariance.  This  representation  is  then  applied  to  derive  the  second  main  result,  which  is 
the  large  deviation  principle  for  a  class  of  Hilbert  space  valued  diffusions  with  small  noise.  This 
result  has  applications  to  the  study  of  image  matching  problems  from  a  Bayesian  perspective, 
and  in  particular  to  the  derivation  of  approximate  maximum  a  posteriori  estimators  in  the  small 
noise  limit. 

[1]  Markov  Chain  Approximations  for  Deterministic  Control  Problems:  Linear  Dynamics  in  the 
Control  and  Quadratic  Cost,  to  appear  in  SIAM  J.  on  Numerical  Analysis. 

[2]  Convergence  of  the  optimal  feedback  policies  in  a  numerical  method  for  a  class  of  deterministic 
optimal  control  problems,  submitted  to  SIAM  J.  on  Control  and  Opt. 

[3]  Second  Order  Numerical  Methods  for  First  Order  Hamilton- Jacobi  Equations,  submitted  to 
SIAM  J.  on  Numerical  Analysis. 

[4]  Convex  Duality  and  the  Skorokhod  Problem,  parts  I  and  II,  to  appear  in  Prob.  Th.  and  Rel. 
Fields. 

[5]  Large  deviations  and  queueing  networks:  methods  for  rate  function  identification,  to  appear 
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in  Stock,  Proc,  and  Their  AppL 

[6]  Simple  necessary  and  sufficient  conditions  for  the  stability  of  constrained  processes,  to  appear 
in  SIAM  J,  on  Applied  Math, 

[7]  Representations  and  large  deviations  for  Hilbert  space  valued  diffusions. 

Part  2.  Work  on  nonlinear  filtering.  We  are  concerned  with  approximations  to  nonlinear 
filtering  problems  that  are  of  interest  over  a  very  long  time  interval  [1].  Since  the  optimal 
filter  can  rarely  be  constructed,  one  needs  to  compute  with  numerically  feasible  approximations. 
The  signal  model  can  be  a  jump-diffusion  or  just  a  process  that  is  approximated  by  a  jump- 
diffusion.  The  observation  noise  can  be  either  white  or  of  wide  bandwidth.  The  observations 
can  be  taken  either  in  discrete  or  continuous  time.  The  cost  of  interest  is  the  pathwise  error  per 
unit  time  over  a  long  time  interval.  It  was  shown  [1]  under  quite  reasonable  conditions  on  the 
approximating  filter  and  the  signal  and  noise  processes  that  (as  time,  bandwidth,  process  and 
filter  approximation,  etc.)  go  to  their  limit  in  any  way  at  all,  the  limit  of  the  pathwise  average 
costs  per  unit  time  is  just  what  one  would  get  if  the  approximating  processes  were  replaced 
by  their  ideal  values  and  the  optimal  filter  were  used.  Analogous  results  are  obtained  (with 
appropriate  scaling)  if  the  observations  are  taken  in  discrete  time,  and  the  sampling  interval 
also  goes  to  zero.  For  these  cases,  the  approximating  filter  is  a  numerical  approximation  to  the 
optimal  filter  for  the  presumed  limit  (signal,  observation  noise)  problem. 

When  suitable  approximating  filters  cannot  be  readily  constructed  due  to  excessive  compu¬ 
tational  requirements  or  to  problems  associated  with  a  high  signal  dimension,  approximations 
based  on  random  sampling  methods  (or,  perhaps,  combinations  of  sampling  and  analytical 
methods)  become  attractive,  and  are  the  subject  of  a  great  deal  of  attention.  This  is  somewhat 
analogous  to  the  use  of  monte  carlo  methods  for  high  dimensional  integration  problems.  Owing 
to  the  sampling  errors  as  well  as  to  the  other  (computational  and  modeling)  approximations  that 
are  made,  in  the  filter  and  signal  processes,  it  is  conceivable  that  the  long  term  pathwise  average 
errors  per  unit  time  will  be  large,  even  with  approximations  that  would  perform  well  over  some 
bounded  time  interval.  The  work  of  the  previous  work  was  extended  [2]  to  a  wide  class  of  such 
algorithms.  Under  quite  broad  conditions,  covering  virtually  all  the  cases  considered  to  date,  it 
was  shown  that  the  pathwise  average  errors  converge  to  the  same  limit  that  would  be  obtained 
if  the  optimal  filter  were  used,  as  time  goes  to  infinity  and  the  approximation  parameter  goes  to 
its  limit  in  any  way  at  all.  It  can  be  extended  to  cover  wide  bandwidth  observation  or  system 
driving  noise  as  well. 

An  effective  form  of  the  Gaussian  or  moment  approximation  method  for  approximating  opti¬ 
mal  nonlinear  filters  with  a  diffusion  signal  process  and  discrete  time  observations  was  developed 
[3],  Various  computational  simplifications  reduce  the  dimensionality  of  whatever  numerical  in¬ 
tegrations  need  to  be  done.  This,  combined  with  an  iterative  Gaussian  quadrature  method 
make  the  filter  effective  for  real  time  use.  The  advantages  are  illustrated  by  a  model  which 
captures  the  general  flavor  of  modeling  the  highly  uncertain  behavior  of  an  object  such  as  a  ship 
near  obstacles  such  as  a  shore  line  into  which  it  cannot  go,  and  must  maneuver  away  in  some 
unknown  fashion.  The  observations  are  of  very  poor  quality,  yet  the  filter  behaves  well  and  is 
quite  stable.  It  performs  than  all  current  alternatives  to  which  it  was  compared,  sometimes 
substantially  better 
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A  large  part  of  the  work  in  communications  an  computer  networks  is  concerned  with  making 
assignments  in  an  optimal  or  very  good  way.  The  work  [4]  was  concerned  with  the  optimal 
control  of  the  assignment  of  jobs  from  several  arriving  random  streams  to  one  of  a  bank  of 
processors.  Owing  to  the  difficulty  of  the  general  problem,  a  heavy  traffic  approach  is  used. 
The  required  work  depends  on  the  processor  to  which  it  is  assigned.  The  information  that  the 
assignment  can  be  based  on  is  quite  flexible,  and  several  information  structures  (data  on  which 
the  control  is  based)  are  considered.  The  assignment  can  be  made  on  arrival  or  when  the  job 
is  to  be  processed.  There  can  be  bursty  arrivals  (the  bursts  depending  on  randomly  varying 
environmental  factors),  rather  general  nonlinear  cost  functions  and  other  complications.  It  was 
shown,  under  reasonably  general  conditions,  that  the  optimal  costs  for  the  physical  systems 
converge  to  the  optimal  cost  for  the  heavy  traffic  limit  problem,  as  the  heavy  traffic  parameter 
goes  to  its  limit.  Numerical  data  is  presented  to  illustrate  some  of  the  potential  uses  of  the  limit 
process  for  obtaining  optimal  controls,  or  controls  satisfying  optimal  tradeoffs  among  competing 
criteria.  The  methods  of  proof  are  quite  powerful  tools  for  such  optimal  control  problems. 

Communications  systems  often  have  many  types  of  users.  Since  they  share  the  same  resource, 
there  is  a  conflict  in  their  needs.  This  conflict  leads  to  the  imposition  of  controls  on  admission 
or  elsewhere.  In  this  work  [5],  there  are  two  types  of  customers,  GP  (Guaranteed  Performance) 
and  BE  (Best  Effort).  We  consider  an  admission  control  of  GP  customer  which  has  two  roles. 
First,  to  guarantee  the  performance  of  the  existing  GP  customers,  and  second,  to  regulate  the 
congestion  for  the  BE  users.  The  optimal  control  problem  for  the  actual  physical  system  is 
difficult.  A  heavy  traffic  approximation  was  used,  with  optimal  or  nearly  optimal  controls.  It 
was  shown  that  the  optimal  values  for  the  physical  system  converge  to  that  for  the  limit  system 
and  that  good  controls  for  the  limit  system  are  also  good  for  the  physical  system.  This  was  done 
for  both  the  discounted  and  average  cost  per  unit  time  cost  criteria.  Additionally,  asymptotically, 
the  pathwise  average  (not  mean)  costs  for  the  physical  system  are  nearly  minimal  when  good 
nearly  optimal  controls  for  the  limit  system  are  used.  Numerical  data  show  that  the  heavy  traffic 
optimal  control  approach  can  lead  to  substantial  reductions  in  waiting  time  for  BE  with  only 
quite  moderate  rejections  of  GP,  under  heavy  traffic.  It  also  shows  that  the  controls  are  often 
linear  in  the  state  variables.  The  approach  has  many  advantages.  It  is  robust,  simplifies  the 
analysis  (both  analytical  and  numerical)  and  allows  a  more  convenient  study  of  the  parametric 
dependencies.  Even  if  optimal  control  is  not  wanted,  the  approach  is  very  convenient  for  a 
systematic  exploration  of  the  possible  tradeoffs  among  the  various  cost  components.  This  is 
done  by  numerically  solving  a  series  of  problems  with  different  weights  on  the  costs.  We  can 
then  get  the  best  tradeoffs,  and  the  control  policies  which  give  them. 

The  work  on  adaptive  stochastic  algorithms  in  communications  [6]  was  motivated  by  some 
general  problems  when  game  theoretic  considerations  enter  in  competitive  problems.  The  follow¬ 
ing  is  a  canonical  problem.  Customers  arrive  at  a  service  center  and  must  choose  between  two 
types  of  service;  a  channel  that  is  shared  by  all  currently  in  it,  and  a  dedicated  line.  The  mean 
service  cost  (or  time)  for  any  customer  entering  the  shared  resource  depends  on  the  decisions 
of  all  future  arrivals  up  to  the  time  of  departure  of  that  customer,  and  so  has  a  competitive 
aspect.  The  system  keeps  a  record  of  the  (discounted)  mean  sample  service  costs  (or  times) 
for  the  shared  resource,  as  a  function  of  the  number  there  when  a  new  arrival  joins,  and  the 
arriving  customers  use  this  to  make  their  decisions.  The  decision  rule  of  each  arriving  customer 
is  based  on  its  own  immediate  self  interest,  given  the  available  data  on  the  past  performance. 
They  select  the  service  with  the  smallest  estimated  service  cost.  But,  if  the  current  estimate 
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of  the  cost  for  the  shared  resource  equals  that  of  the  dedicated  line,  any  decision  is  possible. 
The  procedure  is  a  type  of  learning  algorithm.  The  long  term  behavior  of  the  arrivals  and  its 
effect  on  the  system  averages  is  of  interest.  The  convergence  problem  is  one  in  asynchronous 
stochastic  approximation,  where  the  ODE  might  have  a  set- valued  right  hand  side;  i.e.,  it  might 
be  a  differential  inclusion.  The  set  value  arises  due  to  the  arbitrariness  of  the  decision  at  cer¬ 
tain  values  of  the  current  estimates.  It  was  shown  that,  asymptotically,  the  performance  of  the 
learning  system  is  that  for  the  symmetric  Nash  strategy,  despite  the  allowed  arbitrariness  and 
lack  of  coordination. 
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